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INTRODUCTION 

The  Joint  Strike  Fighter  (JSF)  integrated 
Subsystems  Technology  (J/IST) 
Demonstration  Program  was  a  highly 
successful  industry-wide  five-year  program 
in  which  normally  competitive  JSF  Weapon 
System  Contractors  (WSCs)  and  suppliers 
worked  together  with  competitors  to 
accomplish  technology  maturation  and 
mitigate  technological  risks  of  a 
revolutionary  subsystem  concept  for  tactical 
aircraft  that  integrates  the  subsystems  and 
utility  functions  thereby  reducing  the 
associated  major  equipment  groups  from  13 
to  5.  As  a  result  of  this  integration,  the 
volume,  weight,  and  costs  allocated  to  these 
subsystems  are  significantly  reduced 
without  sacrificing  vehicle  performance  and 
safety.  This  paper  outlines  major  and  minor 
management  and  execution  methods  used 
to  focus,  solidify,  and  maintain  this  industry 
team  over  the  five-year  program.  Details 
about  the  technical  accomplishment  are 
documented  in  the  reports  listed  at  the  end 
of  this  paper. 

BACKGROUND 

In  January  1994,  the  Department  of 
Defense  (DoD)  established  a  joint  service 
organization,  the  Joint  Advanced  Strike 
Technology  (JAST)  Program,  to  define  and 
develop  aircraft,  weapon  system  and 
mission  technologies  for  future  tactical 
aircraft  systems.  The  initial  thrusts  of  this 
organization  were  to  “catalogue  work 
already  underway,  conduct  analysis  of  joint 
service  requirements  and  identify  launch 
candidate  technology  efforts”  (Reference  1). 
The  program  would  establish  technology 


building  blocks  for  the  development  of  the 
next  tactical  aircraft.  Among  the  first  tasks 
of  this  organization  was  to  “focus  and 
integrate”  tactical  aircraft  technology 
programs  and  coordinate  a  “technology 
roadmap*.  To  accomplish  these  objectives 
the  JAST  organization  came  to  the  Air 
Force  and  Navy  Science  and  Technology 
(S&T)  communities  to  identify  what  they 
called  “low  hanging  fruit”  (technologies 
already  under  development  and  had 
creditable  affordability  and  benefits  analysis 
showing  that  the  technology  was  beneficial 
and  the  maturing  of  these  technologies 
could  be  accelerated  with  additional 
funding). 

.  A  series  of  joint  industry  and  government 
meetings  were  held  to  discuss  and  identify 
which  of  the  proposed  technologies  made 
the  most  sense  to  recommend  for 
maturation  in  light  of  established  JAST 
metrics.  These  first  meetings  focused  on 
structuring  a  set  of  unrelated  component 
efforts;  each  one  of  which  would  help  solve 
a  particular  problem,  but  did  not  adequately 
address  the  JAST  metrics. 

The  metrics  of  interest  to  JAST  were  vehicle 
level  benefits  such  as  system  acquisition 
cost,  tri-service  applicability,  sortie 
generation  rate,  and  reduced  logistic 
footprint,  rather  than  the  kind  of  payoffs 
articulated  by  a  single  component  or 
technology  program.  Therefore,  the  idea 
was  bom  to  pull  together  all  of  these 
technologies  or  portions  of  these 
technologies--  whichever  made  sense--and 
enhanced  the  system  level  benefit  in  so 
doing. 

Four  of  the  many  technologies  that 
were  proposed  by  the  S&T  community  for 
JAST  consideration  seemed  ripe  for 
evaluation  as  an  integrated  set.  These 
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technologies  were:  More  Electric  Aircraft 
(MEA)  (References  2, 3,4,5, 6, 7, 8, 9), 
Subsystem  Integration  Technology  (SUIT) 
References  10,11,12,13,14, 15,16), 
Advanced  Vehicle  Management  System 
(A VMS)  (Reference  17),  and  Advanced 
Fixed  Area  Nozzle  (AFAN)  (References 
18,19  sections  4  and  5  respectively).  In 
general  these  four  technology  areas  had 
been  developed  independently,  and  their 
respective  visions  of  integration 
requirements  had  slightly  different  focus.  It 
was  conceivable,  however,  that 
revolutionary  vehicle  level  performance  and 
affordability  benefits  may  be  achievable  in 
the  JAST  time  frame  of  interest  via  an 
integrated  approach  that  would  blend  the 
best  features  of  the  MEA,  SUIT,  AFAN,  and 
AVMS  technology  thrusts.  This  was  the 
premise  of  the  technology  integration  study, 
which  became  known  as  the  Vehicle 
integration  Technology  Planning  Study 
(VITPS).  To  this  end,  in  conjunction  with 
JAST,  four  Wright  Laboratory  Divisions  and 
the  Navy  came  together  to  jointly  manage 
this  JAST  and  Air  Force  funded  study. 

Vehicle  Integration  Technology 
Planning  Study  (VITPS) 

The  VITPS  statement  of  work  (SOW)  was 
written  to  have  Integrated  Product  Teams 
(IPT)  seriously  look  at  all  of  the  technologies 
rather  than  just  favorites,  and  most 
important  to  look  at  ways  to  integrate  the 
best  of  all  of  the  advanced  concepts  and 
technologies.  The  push  was  to  identify  and 
quantify  as  much  as  possible  the  benefits  of 
technology  integration.  Three  independent 
contractor  study  teams  were  formed.  Two 
study  contracts  were  awarded  and  a  third 
JSF  WSC  contractor  decided  to  do  this 
using  Independent  Research  and 
Development  (IRAD)  funds.  These  study 
teams  had  exceptionally  broad  industry 
participation  and  significant  depth,  going 
down  one  or  two  levels  of  suppliers.  This 
extensive  supplier  participation  was 
stimulated  by  the  Request  for  Proposal 
(RFP)  requiring  that  approximately  50%  of 
the  study  resources  go  to  the  suppliers.  The 
government  management  team  selected 
this  approach  since  too  often  the  primary 
study  contractor  reserves  the  majority  of  the 
study  resources  for  itself  and  provides  very 


little  to  its  suppliers.  In  the  case  of  these 
product  areas,  major  weapon  systems 
contractors  procure  this  equipment  from 
suppliers  who  often  take  responsibility  for 
some  integration  work,  and  it  was  essential 
to  have  the  supplier  involvement.  For  the 
VITPS  study,  a  large  supplier  base  was  paid 
to  participate  and  therefore  a  more 
complete  understanding  of  what  happens 
when  technologies  are  integrated  could  be 
assessed  from  the  air  vehicle  down  to 
individual  components. 

These  independent  industry  study  teams 
evaluated  a  broad  range  of  completed  and 
on-going  S&T  and  IRAD  programs  to 
identify  which  technologies,  when 
integrated,  have  potential  to  yield  large 
benefits  in  Life  Cycle  Cost  (LCC)  and  war 
fighting  capability.  Additionally,  an 
assessment  was  made  as  to  which 
integrated  technologies  could  be  matured  by 
CY2000,  CY2005,  and  CY2010  with  some 
reasonable  level  of  resource  investment. 
While  the  analysis  from  each  study  team 
indicated  differences  in  the  exact  level  of 
benefits,  they  were  surprisingly  consistent 
as  to  which  technologies  and  integration 
concepts  enabled  those  benefits 
(References  18,19,20). 

It  was  apparent  to  the  government 
management  team  that  even  though  the 
VITPS  studies  were  being  conducted 
independently  all  three  studies  were 
converging  on  a  similar  end  state 
configuration.  Unfortunately,  each  study 
team  thought  they  had  identified  an 
integration  configuration  that  was  unique 
and  desired  to  keep  it  from  their  competitors 
even  though  two  study  teams  developed 
their  concepts  using  government  funds.  To 
break  this  mind-set,  the  government  team 
pointed  out  that  if  there  was  to  be  a  funded 
technology  maturation  effort  based  on  the 
VITPS  study  results,  it  would  have  to  be  of 
benefit  to  all  three  JAST  WSCs.  There 
were  only  enough  funds  available  to  fund 
one  maturation  program,  and  in  fact,  the 
JAST  model  required  that  technical 
maturation  activities  serve  all  WSCs. 
Therefore,  the  WSCs  were  encouraged  to 
hold  discussions  to  identify  common  ground 
in  their  independently  developed 
configurations. 
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Out  of  these  discussions  it  became  clear 
there  was  enough  common  ground  that  a 
single  technology  maturation  program  could 
serve  all  three  JAST  prime  contractors. 
Additionally,  for  the  JAST  window  of 
opportunity  and  JAST  customers  (Air  Force, 
Navy  and  Marines)  a  common  set  of 
technologies  could  be  integrated  and 
matured  that  would  result  in  vehicle-level 
LCC  and  war  fighting  benefits  that  met  and 
or  exceeded  JAST  metrics.  As  a  result  of 
their  extensive  participation  in  the  VITPS 
studies  the  airframe,  engine  and  subsystem 
contractors  determined  that  this  suite  could 
be  readily  transitioned.  The  three  VITPS 
contractor  study  teams  and  the  government 
management  team  prepared  an  integrated 
advocacy  briefing  to  the  JAST  Program 
Office.  Each  VITPS  contractor  presented 
how  this  technology,  when  matured,  could 
help  accomplish  the  JAST  goals.  The 
VITPS  management  team  presented  its 
approach  for  structuring  and  managing  a 
single  program  that  would  provide  the 
technology  maturation  desired  by  all  three 
WSCs.  Funding  approval  was  then  given  by 
the  JAST  Program  Office  to  prepare  the 
necessary  documentation  for  a  formal 
procurement  solicitation. 


Demonstration  Program  was  its  relationship 
to  the  overall  JAST  Program,  which  in  time 
transitioned  to  a  major  DoD  acquisition 
(ACAT  ID) ,  the  Joint  Strike  Fighter  (JSF) 
Program  (Referenced  21).  The  JAST 
Program  was  a  new  way  of  doing  business, 
bringing  together  war  fighters,  maintainers, 
technologists  and  developers  on  a  single 
joint  service  team  with  a  shared  purpose. 
The  team's  mission  was  to  identify,  mature, 
demonstrate,  and  transition  technologies 
and  concepts  which  meet  war  fighters  needs 
while  reducing  the  cost  of  these  future  joint 
strike  warfare  aircraft  weapon  systems.  To 
accomplish  this  objective,  the  JAST/JSF 
Program  has  three  distinct  phases,  Concept 
Exploration,  Concept  Development,  and 
Concept  Demonstration  &  Flying 
Demonstrators.  The  WSCs  (who  were 
developing  their  Preferred  Weapon  System 
Concepts  [PWSC])  under  this  part  of  the 
JSF  Program  were  viewed  as  the  customer 
of  the  technology  maturation  efforts.  These 
technical  maturation  activities  were  to  run  in 
parallel  with  the  PWSC  development  and 
refinement  as  shown  in  Figure  1.  The 
J/IST  Demonstration  Program  was  one  of 
these  technology  maturation  programs. 


Program  Formulation 

The  JAST  Program  Office  established  a 
joint  AF/NAVY  effort,  the  JAST  Integration 
Subsystem  Technology  (J/IST) 
Demonstration  Program  out  of  what  was  the 
VITPS  government  management  team.  This 
program  was  to  execute  the  necessary 
technical  maturation  activities  and  promote 
transition  of  the  industry  identified 
integrated  subsystem  suite.  The  objective  of 
the  J/IST  Demonstration  Program  was  to 
convince  industry  and  government  decision 
makers  that  the  integrated  subsystem  suite 
provides  promised  LCC  reductions  and 
increased  war  fighting  capability. 
Additionally,  the  program  was  to 
demonstrate  that  the  technology  upon  which 
this  suite  is  based  would  be  sufficiently 
mature  to  enter  E&MD  (Engineering  & 
Manufacturing  Development)  by  the  year 
2000. 

A  major  consideration  in  structuring  the 
management  approach  to  the  J/IST 


JSF  Concept  Demonstration  Phase 


|  FY  94  FY95  FY96  FY97  FY98  FY99  FYOO  FY01  FY02  | 
|  Requirements  EvotutionfCostas  an  Independent  Variable  (C/W)| 


Figure  1  -  JSF  Overall  Program  Schedule 


THE  J/IST  DEMONSTRATION  PROGRAM 

In  January  1995  a  management  team 
composed  of  USAF  and  NAVY  personnel 
was  established  at  the  Air  Force  Research 
Laboratory,  Wright  Patterson  Air  Force 
Base  to  plan  and  execute  the  J/IST 
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Demonstration  Program.  This  team  was 
responsible  for  planning,  program  control, 
financial  management,  and  program 
execution  of  the  J/IST  Demonstration 
Program-subject  to  the  oversight  and 
approval  of  the  JAST  Program  Office. 

Even  though  the  J/IST  Demonstration 
Program  involves  many  technologies  and 
must  deal  with  multi-service  application 
issues,  the  J/IST  Project  Team  was 
intentionally  made  a  small  management 
team  following  the  JAST  model  to  foster 
technology  integration.  The  team  was  made 
up  of  individuals  with  diverse  service  and 
technical  backgrounds  who  are  focused  on 
the  comprehensive  goal  of  transitioning 
integrated  subsystems  rather  than  nurturing 
technology  niches 

One  of  the  first  orders  of  business  for  this 
team  was  to  develop  a  Concept  of 
Operations  (CONOPS)  for  how  this  small 
management  team  would  interface  and  work 
with  industry.  The  primary  customers  were 
the  JSF  WSCs  who  would  “buy"  through 
incorporation  in  their  JSF  weapon  system 
designs,  the  product  of  the  J/IST 
Demonstration  Program.  Traditionally,  the 
customer  for  a  technology  maturation  effort 
is  the  government  project  office  that  funds 
and  executes  the  effort.  This  government 
project  office  would  set  the  requirements, 
represent  the  interests  of  the  JAST  Program 
Office  and  act  as  an  “honest  broker"  among 
the  executing  WSCs  who  were  in  fierce 
competition  to  win  the  Joint  Strike  Fighter 
development  contract.  From  the  start,  the 
J/IST  Management  Team  realized  that  their 
role  would  to  be  to  act  more  like  program 
shepherds  than  requirement  givers.  This 
realization  emerged  from  the  following 
facts.  The  VITPS  study  identified  a 
subsystem  integration  concept  to  be 
matured,  but  did  not  identify  any 
requirements  as  to  size,  performance 
metrics,  etc.  Secondly,  the  JSF  WSCs,  who 
were  developing  their  highly  proprietary 
aircraft  design,  needed  the  J/IST 
Demonstration  Program  to  provide  value  to 
them  without  giving  away  their  proprietary 
concepts.  Additionally,  the  PWSC  concepts 
would  be  evolving  throughout  the  duration 
of  the  J/IST  Demonstration  Program  so  that 
the  requirements  that  defined  the  J/IST 
program  had  to  be  robust  enough  to  permit 


the  J/IST  Program  to  be  relevant  even  after 
several  design  iterations. 

To  ensure  a  high  probability  of  transition  of 
J/IST  technologies,  the  J/IST  Program 
established  an  Industry/Customer  User 
Team  (ICUT)  and  invited  the  five  air 
framers,  three  jet  engine  manufactures,  and 
government  representatives  to  participate. 

In  establishing  the  ICUT,  the  contractors 
were  told  that  J/IST  Demonstration  Program 
would  be  competitively  awarded  to 
accomplish  demonstrations  that  address  the 
transition  risk  issues  identified  by  the  ICUT. 
All  members  of  the  ICUT  not  awarded 
specific  demonstrations  had  the  opportunity 
to  take  part  in  the  technical  review  and 
assessment  team  that  would  closely  track 
the  progress  and  results  from  this  program. 


J/IST  TECHNOLOGY  DEMONSTRATION  CONSENSUS 
ARCHITECTURE 


Figure  2  J/IST  BAA  95-3  Demonstration 
Consensus  Architecture 


To  start  the  discussions,  the  J/IST 
Management  Team  critically  evaluated  the 
results  of  the  three  VITPS  study  teams  and 
then  and  condensed  to  6  essential 
technology  sets  or  integrations.  A 
document  was  prepared  that  captured  a 
“consensus  architecture"  that  encompassed 
the  6  essential  technology  sets  or 
integrations.  The  initial  draft  of  this 
document,  the  Consensus  Technology 
Package  (CTP),  had  the  J/IST  Management 
Team’s  perspective  as  to  the  transition  risk 
issues  that  needed  to  be  addressed  in  the 
J/IST  Demonstration  Program  in  order  to 
reduce  the  transition  risk  to  low  for  E&MD. 
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The  ICUT  and  selected  government 
technical  experts  reviewed  this  document 
and  after  several  iterations  a  final  version 
emerged.  This  iterative  process  was 
facilitated  in  that  the  initial  draft  of  the  CTP 
plainly  stated  that  the  forthcoming  request 
for  proposals  (RFP)  for  the  J/IST 
Demonstration  Program  would  be  a 
competition  on  methodology:  the 
methodology  to  cost  effectively  demonstrate 
the  “consensus  architecture”  and  reduce  to 
low,  the  identified  transition  risks  outlined  in 
the  CTP.  Novel  ways  of  integrating,  while 
interesting  were  not  to  be  used  as  source 
selection  criteria.  The  resultant  CTP  was 
included  as  part  of  the  Joint  Advanced 
Strike  Fighter  Technology  Broad  Agency 
Announcement  95-3,  issued  by  the  Program 
Office  on  April  7, 1995.  This 
Announcement  provided  data  to  assist  the 
bidder  in  proposal  preparation.  The 
consensus  architecture  that  was  part  of  the 
BAA  is  shown  in  Figure  2. 

This  approach  enabled  the  J/IST  customers 
to  make  sure  that  their  issues  and  concerns 
that  they  chose  to  share  would  be 
addressed  in  the  forthcoming  procurement. 
The  focus  was  on  capturing  technical  issues 
and  concerns  rather  than  the  generation  of 
detailed  design  requirements  for  the 
demonstration  hardware.  This  would  allow 
a  bidder  to  be  creative  in  using  existing 
facilities  and/or  equipment  that  could  work 
the  issues  and  not  be  viewed  by  other 
contractors  as  working  their  prototype 
hardware  configurations  for  their  proprietary 
aircraft  design.  Any  such  perception  would 
have  destroyed  willingness  to  work  together 
and  share  information. 

The  J/IST  Demonstration  Program  was 
planned  to  be  a  single  program  that  reduces 
to  low  the  transition  risk  for  E&MD  of 
integrated  subsystems  technology  for  all 
JSF  WSCs.  The  consensus  technology 
package  captured  the  issues  and  concerns 
that  needed  to  be  addressed  in  the  J/IST 
Demonstration  Program;  however,  every 
airframe  contractor  could  not  be  selected  to 
accomplish  every  demonstration  due  to 
funding  limitations.  Therefore,  the 
evaluation  criteria  for  BAA  95-3  clearly 
stated  a  major  evaluation  factor  was  the 
extent  to  which  the  bidder  and  the  proposed 


demonstration  plan  would  provide 
meaningful  sharing  of  progress,  results,  and 
lessons  learned  with  the  other  JSF  WSCs. 
Additionally,  all  bidders  were  required  to 
propose  demonstrations  that  addressed  all 
elements  of  the  J/IST  consensus 
architecture.  This  was  done  so  that  for 
each  major  technology  set  there  would  be 
several  different  demonstration  approaches 
to  choose  from.  This  would  allow  the 
source  selection  team  to  weave  together  the 
best  mix  of  contractors  and  demonstrations. 
To  facilitate  this  approach,  cost  and 
technical  effort  had  to  be  proposed  in  easily 
identifiable  and  severable  tasks. 


In  structuring  BAA  95-3  it  was  recognized 
also  that  the  J/IST  Demonstration  Program 
had  to  address  two  categories  of  issues  - 
technical  and  those  related  to  affordability. 
Two  general  types  of  tasks  were  called  out 
in  the  BAA.  One  task  was  to  reduce  to  low 
the  transition  risk  of  this  technology.  The 
other  task  was  to  accomplish  a  more  refined 
affordability  assessment  using  the  results  of 
the  various  J/IST  technical  activities. 

To  further  the  likelihood  that  JSF  WSC 
contractor  issues  and  concerns  were 
worked,  the  BAA  required  the  proposed 
program  to  be  divided  into  three  phases: 
Phase  1-  Program  Definition  and  Planning, 
Phase  2  -  Detail  Demonstration  Hardware 
Design  and  Phase  3  -  Hardware  Fabrication 
and  Demonstration  Execution.  The 
beginning  and  ending  of  these  three  phases 
were  key  milestones  at  which  go/no  go 
decisions  could  be  made. 

From  the  proposals  received,  contractors 
were  selected  to  accomplish  major  tasks 
and  demonstrations.  The  contracts  from  the 
BAA  solicitation  contained  contract 
language  that  designated  the  contractor 
doing  a  particular  task  or  demonstration  as 
“a  Responsible  Agent  Contractor”  for  the 
other  JSF  WSC  contractors.  As  a 
Responsible  Agent  Contractor,  it  was 
understood  that  the  executing  contractor 
was  doing  the  work  for  the  other  two  JSF 
WSC  contractors  and  the  government,  as 
well  as  for  their  own  company.  To  facilitate 
this,  Associate  Contractor  Agreement 
clauses  and  data  sharing  tasks  were  written 
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into  these  contracts.  This  would  allow 
direct  sharing  of  technical  information 
among  the  contractors  and  eliminate  the 
government  team  being  a  choke  point  for 
this  sharing. 


TECHNICAL  PROGRAM 
PLANNING 

Program  planning  was  envisioned  as  Phase 
1 ,  a  four-month  effort,  for  the  J/IST 
Demonstration  program.  It  turned  out  that 
program  planning  occurred  throughout  the 
entire  program.  During  Phase  1 ,  the  J/IST 
suite  concept  and  the  innovative 
demonstration  approaches,  methodologies, 
and  techniques  selected  in  source  selection 
were  converted  into  comprehensive  plans 
for  implementation  of  the  concept  and 
demonstration  of  the  critical,  enabling 
technologies  so  that  technically  sound  and 
affordable  low  risk  transition  into  E&MD 
could  be  assured.  This  phase  would  allow 
contractors  who  were  not  selected  to  be  the 
responsible  agent  contractor  for  a  particular 
series  of  demonstrations  to  make  sure  their 
issues  and  concerns  were  adequately 
addressed. 

At  the  initial  J/IST  Phase  1  planning 
meeting,  the  J/IST  management  team  made 
it  very  clear  that  Independent  Product 
Teams  (IPTs),  including  the  J/IST 
Responsible  Agent  Contractors  and  JSF 
WSC  design  engineers,  would  be  the 
forums  to  dovetail  planning,  review  of 
demonstration  plans,  and  track/assess 
technical  progress.  Members  of  the  J/IST 
management  team  would  sit  in  on  the  IPT 
deliberations  as  facilitator/observers  and  to 
make  sure  that  contractual  scope  was  not 
being  violated.  Three  IPTs  were  formed  to 
execute  the  J/IST  Demonstration  Program. 
These  IPTs  were  Power  and  Actuation 
(P&A),  Thermal  Energy  Management 
Module  (T/EMM),  and  Integration  and 
Analysis  (l&A).  The  Responsible  Agent 
Contractor  who  was  primarily  responsible  for 


the  bulk  of  the  demonstrations  or  technical 
activities  that  fell  within  each  IPT 
technology  domain  was  selected  to  lead  and 
document  the  IPT  deliberations  and 
summarize  the  recommendations.  The  l&A 
IPT  was  led  by  the  J/IST  Management 
Team  since  the  l&A  IPT  was  not  responsible 
for  executing  demonstrations  but  rather 
ensuring  that  everything  was  integrated  to 
produce  the  desired  results. 

At  first  it  was  difficult  for  the  two 
demonstration  IPTs,  T/EMM  and  P&A,  to 
solidify.  The  apparent  reason  for  this  was 
the  significantly  different  experience  base  of 
the  engineers  from  the  three  JSF  WSCs. 

For  example,  while  they  all  wanted  to  work 
many  of  the  same  issues,  the  level  of 
understanding  or  knowledge  about  the 
existing  state-of-the-art  around  those  issues 
varied  greatly.  An  engineer  with  limited 
experience  with  the  issue  wanted  to  use 
J/IST  resources  to  work  technology 
maturation  concerns  that  have  been  already 
resolved  by  other  technical  programs.  This 
was  overcome  by  a  contractor  initiative  to 
share  with  the  IPT  the  technical  content  of 
their  own  proposal  they  had  submitted  for 
the  BAA  95-3  competition.  This  provided  a 
framework  for  everyone  on  the 
demonstration  IPTs  to  come  to  a  common 
level  of  understanding  concerning  the 
various  perspectives  on  a  given  issue  and 
the  state-of-the-art  for  a  particular 
technology.  It  had  the  additional  benefit  of 
establishing  a  level  of  trust  among  the 
program  participants. 

DETAILED  REQUIREMENTS 
DEFINITION  AND  RISK  TRACKING 

The  product  from  Phase  1  of  the  J/IST 
Demonstration  Program  was  a  robust  set  of 
integrated  design  and  demonstration 
requirements,  along  with  a  risk  reduction 
tracking  scheme  for  the  rest  of  the  program. 
The  process  to  develop  requirements  began 
before  the  release  of  BAA  95-3.  The 
activity  to  develop  and  establish  a 
consensus  technology  package  that  was 
part  of  the  BAA  was  the  initial  step.  During 
Phase  1 ,  the  top-level  notional  consensus 
architecture  shown  in  Figure  2  was  evolved 
into  a  more  detailed  engineering  functional 
configuration  for  demonstration  purposes 
using  the  process  outlined  in  Figure  3. 
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From  an  overview  perspective,  all  who 
proposed  to  the  J/IST  BAA  proposed  two 
major  technology  maturation  paths.  One 
path  was  a  series  of  comprehensive  ground 
demonstrations  while  the  other  path 
involved  ground  demonstrations  followed  by 
flight-testing.  These  were  judged  to  be 
acceptable  and  cost  effective  ways  to 
mature  the  integrated  J/IST  technology 
suite.  To  accomplish  this,  the  consensus 
demonstration  architecture  shown  in  Figure 
2  evolved  into  two  major  demonstration 
architectures  -  one  for  those  technology 
that  could  be  matured  doing  only  ground- 
based  demonstrations  and  another  for  those 
technologies  that  required  both  ground 
demonstrations  and  flight-testing.  Where 
practical  and  cost  effective,  the  same 
component  elements  were  used  for  both 
technology  maturation  paths.  This  involved 
some  adjustments  to  a  few  of  the  elements 
originally  proposed  by  the  bidders. 

Each  WSC  came  to  the  Demonstration  IPTs 
with  independently  developed  inputs 
including  lists  of  key  parameters  and 
performance  ranges  they  wanted 
demonstrated  or  measured.  On  the  whole, 
these  independently  developed  inputs  were 
compatible,  but  there  were  areas  of 
conflicting  requirements  and  desires.  To 
arrive  at  a  set  of  robust  design  requirements 
for  a  given  demonstration,  hardware  mini¬ 
trade  studies  and  analyses  were 
accomplished.  These  analyses  had  to  take 
into  account  both  the  desires  of  the  WSCs 
and  the  hard  constraints  imposed  by  the 
already  purchased  demonstrations.  For 
example,  items  to  be  flown  had  to  fit  and  be 
compatible  with  the  F-16  while  items  that 
had  to  interface  with  an  F-119  engine  had  to 
be  compatible  with  it.  Extensive 
modifications  to  either  the  F-16  or  the  F-119 
engine  were  out  of  the  question  due  to 
financial  constraints. 


The  min-trades  and  analyses  involved  an 
iterative  process  to  balance  WSC  desires, 
J/IST  hard  demonstration  constraints,  laws 
of  physics,  and  available  resources.  The 
Responsible  Agent  Contractor  and  his  team 
accomplished  engineering  analysis  to 
identify  a  set  of  design  parameters  that 
would  result  in  demonstration  hardware  that 


could  provide  data  for  as  many  of  the  WSC 
identified  issues  as  possible  while  still  being 
compatible  with  all  of  the  other  constraints. 
This  involved  detailed  engineering  analysis 
followed  by  an  across  the  board  review  of 
all  the  issues  and  concerns  to  see  how 
many  were  not  addressed  appropriately. 
This  iterative  process  eventually  converged 
to  a  final  set  of  demonstration  hardware 
requirements  that  the  IPT  felt  would  allow 
the  planned  demonstration  to  appropriately 
address  all  of  the  identified  issues  and 
concerns.  This  iterative  process  had  an 
additional  benefit  as  the  JSF  WSCs  gained 
insight  into  what  were  the  key  metrics  or 
parameters  that  drive  the  design.  This 
knowledge  often  resulted  in  a  WSC 
restating  their  concern  or  issue  in  ways  that 
still  provided  them  the  key  knowledge  and 
data  they  wanted  but  enabled  one 
demonstration  hardware  design  to  work  all 
of  the  issues  and  concerns. 

The  actual  demonstrations  were  conducted 
using  hardware  that  worked  the  risk  issues 
rather  than  building  flight-sized  or  prototype 
hardware.  Flight-sized  hardware  was 
fabricated  for  those  components  where  risk 
issues  involved  size  and  weight  in  relation 
to  performance  level.  Commonly,  laboratory 
or  industrial  equipment  was  used  for 
features  such  as  lubrication  systems  or 
power  supplies.  This  kept  the  focus  on  the 
primary  purpose  of  the  demonstration, 
which  was  to  gather  data  on  the  key  risk 
elements  of  the  architecture 
To  facilitate  this  technology  maturation 
process,  a  tracking  methodology  was  used 
to  insure  all  issues  and  concerns  were 
addressed.  Figure  4  shows  the  three  charts 
used  to  capture  this  tracking  methodology. 

A  compliance  matrix  called  the  Maturation 
Matrix  (MoM)  was  constructed  that  lists,  for 
each  milestone,  the  WSC  issue  or  issues 
that  would  be  addressed  by  data,  and 
results  produced  by  completion  of  that 
milestone.  A  milestone  could  be  the 
completion  of  a  detailed  analysis,  passing 
preliminary  and  critical  design  reviews  ( 

PDR  &  CDR),  or  completion  of  a 
demonstration  or  test.  Prior  to  any  technical . 
work  the  anticipated  amount  of  risk 
reduction  available  from  each  milestone  for 
each  WSC  issue  was  assessed  by  the 
appropriate  IPT.  Based  upon  this 
assessment,  a  second  tracking  chart,  the 
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“Key  Element  Risk  Tracking  (KERT)”  chart,  that  tracks  the  highest  risk  level  for  any 

was  structured  to  track  the  amount  of  risk  element  of  the  J/IST  Program.  The 

reduction  anticipated  by  the  IPT  from  each  waterfall  chart  shows  the  anticipated  overall 

demonstration  milestone.  From  the  KERT  risk  reduction  progression  as  a  function  of 

chart,  a  risk  waterfall  chart  was  constructed  demonstration  milestone. 


Figure  3  -  Requirements  Generation  and  Technology  Maturation  Process  Flow 
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Figure  4  Technology  Maturation  and  Risk  Tracking  Charts 
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Figure  5  -  IPT  Risk  Reduction  Planning  and  Verification  Process 
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Figure  5  outlines  the  extensive  participation 
of  the  IPTs  in  the  planning  and  execution  of 
the  demonstrations.  It  also  highlights  IPT 
involvement  in  the  assessment  and  tracking 
of  the  risk  reduction  achieved  from  each 
demonstration.  The  three  crosshatched 
blocks  represent  the  activities  accomplished 
primarily  during  Phase  1  of  the  J/IST 
Demonstration  Program.  The  rest  of  the 
blocks  represent  the  activities  before  and 
after  accomplishment  of  each  milestone. 
After  a  milestone  event  was  completed,  the 
Responsible  Agent  Contractor  who 
conducted  it  prepared  a  coordination  memo 
documenting  the  results  of  the  milestone 
event.  The  appropriate  J/IST  IPT  would 
evaluate  this  document,  technical  data  and 
results  of  the  witnessed  demonstration  to 
determine  if  the  desired  level  of  risk 
reduction  was  achieved  or  to  identify  any 
shortfalls. 

If  the  milestone  event  produced  risk 
reduction  that  met  or  exceeded  the 
anticipated  values,  the  JSF  WSCs  design 
team  representatives  signed  the 
coordination  memo.  If  the  accomplished 
level  of  risk  reduction  was  more  than 
expected,  the  KERT  and  waterfall  charts 
were  adjusted  to  accurately  reflect  the 
actual  achieved  level  of  risk  reduction. 
Conversely,  if  the  level  of  risk  reduction  was 
less  than  anticipated,  an  evaluation  to 
assess  the  impact  of  this  shortfall  was 
performed.  This  assessment  examined  the 
impact  on  the  J/IST  critical  path  leading  to 
the  overshadowing  demonstrations. 

The  overshadowing  demonstrations  were 
those  functions  that  the  J/IST  customers 
indicated  had  to  be  demonstrated  in  order  to 
transition  this  technology  to  their  JSF 
aircraft  designs.  The  J/IST  customers 
identified  two  overshadowing 
demonstrations:  (1)  Starting  the  F-1 1 9 
engine  using  the  integrated  subsystem  suite 
during  the  integrated  engine  and  integrated 
subsystem  ground  demonstration  and  (2) 
flying  the  F-1 6  with  all  primary  flight  control 
surfaces  powered  electrically. 

If  the  shortfall  was  not  on  the  critical  path 
then  it  was  just  noted  and  the  program 
continued  as  planned.  If  the  critical  path 
was  impacted,  the  Responsible  Agent 
Contractor  for  that  demonstration  milestone 


proposed  how  the  risk  reduction  shortfall 
could  be  eliminated.  This  proposal  could 
include  additional  testing,  analysis  or 
modifying  future  demonstrations.  The 
executing  IPT  evaluated  the  technical 
merits  of  the  proposal  while  the  government 
management  team  and  executing  contractor 
evaluated  the  financial  implications  of  the 
proposal. 

If  the  proposed  modified  work-plan  had  no 
net  cost  impact  and  would  solve  the  risk 
reduction  shortfall,  the  plan  would  be 
implemented.  If,  however,  the  only  way  of 
filling  in  the  shortfall  required  technical 
and/or  programmatic  decisions  with  cost  or 
schedule  impacts  to  major  program 
milestones  or  involved  significant  deviations 
from  the  demonstration  consensus 
architecture,  the  ICUT  became  involved. 

The  ICUT  transitioned  into  what  became  the 
J/IST  Executive  Council.  The  name  change 
from  ICUT  to  J/IST  Executive  Council 
reflects  the  changing  role  of  this  group,  a 
shift  in  membership,  and  to  provide 
additional  prestige  so  that  the  appropriate 
decision  makers  needed  at  these  meetings 
would  participate.  This  council  was  made  up 
of  the  JSF  Program  Office  Flight  Systems 
IPT  Lead,  JSF  WSC  subsystem  design 
teams,  J/IST  Management  Team,  and  the 
program  managers  from  each  of  the  J/IST 
contracts.  If  a  particular  demonstration  had 
a  shortfall  in  achieving  risk  reduction,  this 
council  reviewed  the  associated  technical 
issues  and  the  recommended  solutions) 
proposed  by  the  Responsible  Agent 
Contractor.  A  particular  corrective  action 
was  not  implemented  unless  this  council 
agreed  that  the  approach  still  provided 
value  to  the  JSF  WSCs,  and  the  potential 
level  of  risk  reduction  from  this  approach 
was  acceptable. 

In  all  cases  the  financial  implications  of  any 
technical  approach  were  never  presented  to 
or  discussed  by  the  J/IST  Executive 
Council.  All  financial  discussions  were 
private  between  the  J/IST  Management 
Team  and  the  effected  responsible  agent 
contractor.  All  the  executive  council  knew 
was  the  viability  of  the  proposed  approach 
within  current  the  J/IST  budget,  and  whether 
it  was  within  the  contract  scope  of  the 
effected  contractor.  If  the  risk  reduction 
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shortfall  could  not  be  corrected  within  the 
available  budget  the  J/IST  Executive 
Council  identified  lower  priority  tasks  that 
could  be  sources  of  funding. 


AFFORDABILITY  ASSESSMENT 
PLANNING 


The  affordability  assessment  task  within  the 
J/IST  Demonstration  Program  was  a  major 
activity  that  in  its  final  form  required  close 
coordination  and  cooperation  among  the 
three  JSF  WSCs.  Initially,  all  three  JSF 
WSCs  were  contracted  to  work 
independently-essentially  rerunning  the  just 
completed  VITPS  type  of  benefits 
assessments.  Each  JSF  WSC  would  be 
using  their  VITPS  generic  study  aircraft  and 
methodology  just  like  in  the  VITPS  studies. 
This  was  viewed  to  be  a  low  cost  effort 
since  the  WSCs  had  just  completed  the 
VITPS  studies,  and  for  the  most  part,  all 
that  had  to  be  done  was  to  rerun  these 
studies  using  a  more  refined  set  of 
component  data  produced  during  the 
various  J/IST  technical  activities.  The 
perceived  value  of  this  activity  was 
validation  of  whether  the  projected  benefits 
from  the  VITPS  studies  were  maintained  or 
enhanced  as  the  technology  maturation 
activities  produced  data. 

During  the  Program  Definition  and  Planning 
Phase,  Phase  1  of  the  J/IST  Demonstration 
Program,  the  JSF  WSCs  voiced  concern 
that,  from  their  perspective,  there  was  not 
much  value  in  reaccomplishing  the  VITPS 
studies.  The  benefit  assessments  of  great 
value  to  them  were  those  being  made  on 
their  proprietary  JSF  competition  aircraft  as 
the  J/IST  technology  was  folded  into  these 
designs.  The  resultant  studies  obviously 
would  be  proprietary,  but  they  saw  the  J/IST 
Demonstration  Program  providing  needed 
refined  data  that  they  could  use  in  these 
proprietary  studies.  Refined  data  of  interest 
would  be  the  emerging  results  of  the  various 
J/IST  technology  maturation  efforts. 
Therefore  a  VITPS-like  study  as  part  of  the 
J/IST  Demonstration  Program  seemed  of 
little  value. 


During  the  Phase  1  discussions  concerning 
keeping  or  eliminating  the  contracted 
benefit  assessment  studies,  another 
perspective  emerged.  The  JSF  Program 
Office  was  developing  the  Joint  Logistic 
Modeling  Environment  (JLME)  that  would 
eventually  be  used  to  help  evaluate  the 
proprietary  aircraft  designs  for  JSF  E&MD 
selection.  It  seemed  that  if  J/IST,  utilizing 
JLME,  did  an  open-to-all  WSCs  benefits 
assessment  within  the  J/IST  Demonstration 
Program,  it  would  provide  an  opportunity  to 
exercise  this  new  environment  where 
problems  could  be  identified  and  worked  out 
before  use  in  the  proprietary  arena.  This 
activity  would  only  need  one  assessment 
study  using  resources  initially  allocated  to 
accomplish  the  original  three  studies. 

Drawings  of  the  candidate  (non-proprietary) 
study  aircraft  were  exchanged  and  reviewed 
by  the  l&A  IPT  and  JSF  WSCs  to  identify 
which  aircraft  design  should  be  used  for  the 
study.  The  consensus  was  that  the  Boeing 
Seattle  study  aircraft  was  the  best  vehicle 
for  this  study  based  upon  level  of  technical 
detail  already  available.  Boeing  Seattle  was 
selected  to  crunch  the  numbers,  with 
McDonnell  Douglas  (now  Boeing  St.  Louis) 
and  Lockheed  Martin  providing  the  detailed 
parameters  from  their  J/IST  component 
suppliers.  The  list  of  components  was 
divided  and  the  suppliers  designed-through 
to  least  a  PDR  level  of  detail-components 
for  both  a  traditional  federated  subsystem 
and  a  J/IST  integrated  subsystem 
configuration.  The  same  supplier  designed 
both  the  federated  and  J/IST  components 
for  the  same/similar  function  to  reduce 
unavoidable  bias.  The  consensus 
architecture  and  requirements  were  used  to 
size  and  configure  the  J/IST  study 
components.  The  consensus  requirements 
and  an  F-22-like  subsystem  architecture 
were  used  to  size  and  configure  the 
traditional  federated  components. 

A  benefit  assessment  involves  more 
information  than  just  component 
engineering  parameters.  Additional  studies 
were  accomplished  (Figure  6)  to  get  a 
handle  on  some  of  these  other  factors. 

Virtual  installation  and  supportability  studies 
developed  typical  maintenance  parameters 
and  ground  support  equipment  requirements 
for  J/IST  subsystem  (Reference  22).  A 
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quick-look  study  was  also  conducted  that 
looked  at  the  impact  of  J/IST  integrated 
subsystem  on  ground  and  carrier  support 
requirements  (Reference  23).  The  flow  of 
information  from  these  supporting  studies  to 
the  benefit  assessment  study  (Reference 
24)  is  shown  in  Figure  6. 


Affordability  Assessments 


Figure  6  Flow  of  Affordability  Information 

COMMUNICATION  &  TEAM 
BUILDING 

Good,  open,  and  frequent  communication  is 
critical  to  team  building  and  the  success  of 
any  large  project  involving  many  different 
players.  This  was  especially  important  to  the 
success  of  the  J/IST  Demonstration 
Program,  since  the  J/IST  Demonstration 
Program  had  to  resolve  the  issues  and 
concerns  of  the  JSF  WSC  contractorswho 
were  engaged  in  a  fierce  competition  to  win 
the  JSF  Program  contract.  To  consolidate 
these  three  voices  into  one,  the  J/IST 
management  team  used  several 
mechanisms  and  techniques  outlined  below: 

(1)  The  J/IST  contracts  were  written 
with  clear  delineation  of  technical 
task  responsibilities  so  that 
competition  for  doing  J/IST  tasks 
ended  with  the  source  selection 
decision.  If  the  scope  of  a  technical 
task  had  to  change  during  the 
execution  of  the  J/IST  program, 
these  changes  were  made  through 
the  contractor  responsible  for  that 
task. 


(2)  Team  administrative  tasks  were 
distributed  among  J/IST  contracts. 
The  Lockheed  Martin  Team  was 
responsible  for  making  finalized 
viewgraphs  and  charts  for  joint 
presentations.  The  Boeing  St.  Louis 
Team  was  responsible  for 
constructing  the  J/IST  Master  Plan 
and  risk  tracking  documentation. 

(3)  The  Boeing  Seattle  and  Lockheed 
Martin  contracts  contained  tasks  to 
provide  inputs  to  the  development 
of  the  Master  Plan  being  assembled 
by  Boeing  St.  Louis. 

(4)  The  “Responsible  Agent 
Contractor”  concept  was  written  into 
each  contract.  A  “Responsible 
Agent  Contractor”  was  the 
contractor  under  contract  to 
accomplish  a  given  task  for  the 
entire  J/IST  Team. 

(5)  Each  J/IST  contract  had  the 
requirement  to  establish  Associate 
Contractor  Agreement  “ACA”, 
clauses  to  enable  direct  information 
flow  between  all  participating  J/IST 
contractors.  This  avoided  the 
problem  of  needing  to  funnel  data 
through  the  government  J/IST 
Demonstration  Team. 

(6)  The  J/IST  Demonstration  Team  and 
the  executing  contractors  jointly 
briefed  technical  plans  and  progress 
to  the  JSF  Program  Office.  These 
were  unified  briefs,  not  contractor 
centric.  It  was  common  that  one 
contractor  would  brief  technical 
work  being  accomplished  by 
another  contractor  since  they  were 
both  working  on  different  aspects  of 
the  same  issues,  and  the  work 
logically  fit  together  for  presentation 
continuity. 

(7)  Weekly  "meet  me"  calls  were  held 
for  each  J/IST  IPT.  These  weekly 
calls  were  standing  meetings  with 
written  minutes.  To  facilitate 
discussions,  copies  of  drawings, 
viewgraph  charts  or  test  plans  were 
E-mailed  ahead  of  time  to  the  team 
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members  so  that  everyone  had 
copies  in  front  of  them  during  these 
discussions.  If  the  files  were  large, 
sensitive,  or  contained  proprietary 
information,  they  were  uploaded 
onto  a  secure  FTP  site  operated  by 
the  JSF  Program  Office  or  E-mailed 
in  an  encrypted  form.  Often 
additional  telephone  conference 
calls  were  set  up  to  hold  in-depth 
discussions  or  reviews  of  specific 
technical  issues.  This  approach 
saved  the  J/IST  program  travel 
costs  and  loss  of  productive  man¬ 
hours  during  travel. 

(8)  All  J/IST  contractors  participated  in 
the  demonstration  hardware 
preliminary  and  critical  design 
reviews  (PDR  &  CDR).  The 
tendency  by  some  contractors  to 
stamp  almost  everything  as 
proprietary  was  strongly  challenged 
by  the  government  team.  If 
necessary,  short,  closed-door 
limited  audience  discussions  were 
held.  But,  for  the  most  part,  PDRs 
and  CDRs  were  open  meeting  since 
eventually  all  of  this  hardware  had 
to  work  together  in  the  various 
demonstrations. 

(9)  The  role  of  government  technical 
experts  is  best  summed  up  by  the 
phrase,  “insight  not  oversight," 
providing  wise  council  from  their 
expertise  and  experience,  but  not 
direction  and/or  establishment  of 
requirements.  While  most 
government  experts  understood  and 
performed  this  role,  there  was  a 
natural  tendency  on  the  part  of 
some  of  the  contractors  to  look  to 
the  government  IPT  members  for 
more. 

(10)  Major  programmatic  decisions  that 
emerged  from  program  milestone 
shortfalls  were  elevated  to  the  J/IST 
Executive  Council  for  resolution. 

This  council  membership  was  made 
up  of  the  JSF  Program  Office  Flight 
Systems  Lead,  JSF  WSC 
subsystem  design  teams,  J/IST 
Management  Team,  and  the 


program  managers  from  each  of  the 
J/IST  contracts. 

(11)  The  entire  team  got  together  about 
every  6  months  -  initially  for 
program  planning,  PDRs,  and  CDRs 
But  when  the  program  moved  into 
Phase  3,  technical  interchange 
meetings  (TIMs)  were  scheduled. 
Technical  interchange  meetings 
were  contracted  meetings  for  the 
last  three  years  of  the  program  as 
forums  where  the  executing 
contractor  team,  the  “Responsible 
Agent  Contractor,”  presented  the 
technical  progress  for  the  past  six 
months  to  the  rest  of  the  JSF  WSC 
and  J/IST  contractor  community. 
Additionally,  the  executing 
contractor  presented  planned 
technical  activity  for  the  next  9 
months.  These  meetings  were 
technical  workshops  with  open 
exchanges  among  the  technical 
experts,  which  resulted  in  extensive 
technical  interchange.  The 
presentation  of  the  planned 
technical  activities  for  the  next  nine 
months  was  to  inform  and  invite 
JSF  WSCs  and  J/IST  contractor 
involvement  in  upcoming  efforts  if 
they  so  chose.  Generally, 
participation  involved  special 
analyses,  reviewing,  and  providing 
comments  on  test  plans.  Early  on 
in  the  program,  the  suppliers  were 
hesitant  about  briefing  their 
technical  work  with  competitors  in 
the  room.  Often,  a  supplier  or  two 
had  to  leave  so  that  a  proprietary 
brief  could  be  made.  But,  as 
technical  results  became  available, 
the  level  of  sensitivity  decreased 
until  a  proprietary  briefing  was  the 
exception  rather  than  the  rule.  One 
got  the  impression  that  they  wanted 
to  broadcast  their  technical 
accomplishments. 

(12)  In  the  initial  phase  of  the  program 
face-to-face  meetings  were 
commonplace  and  encouraged.  It 
helps  to  foster  good  working 
relationships  if  individuals  have  met 
and  developed  a  relationship.  This 
helped  to  build  a  team  spirit  since 
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you  got  to  meet  and  know  your 
technical  counterparts  in  the 
industry.  To  nurture  this  J/IST  team 
spirit,  after  business  hours  social 
events  were  commonly  scheduled 
in  conjunction  with  IPT  meeting  and 
major  program  reviews.  These 
events  ranged  from  cookouts  to 
dinners  together  at  a  local 
restaurant  with  local  character. 
These  were  always  arranged  and 
paid  for  by  the  individual,  never  at 
contract  expense.  As  the  program 
progressed,  the  approach  of  choice 
for  IPT  meetings,  technical  planning 
and  the  Executive  Council  were 
electronic  meetings  via  meet  me 
telephone  calls  with  pre-distributed 
documentation.  At  the  end  of  the 
program,  face-to-face  meetings 
were  generally  only  to  witness  a 
J/IST  major  milestone  or  to  attend  a 
technical  interchange  meeting 
(TIM). 


SUMMARY 


The  JSF  Integrated  Subsystem  Technology 
(J/IST)  Demonstration  Program  was  a  highly 
successful  technology  maturation  program 
that  made  available  to  all  three  JSF  aircraft 
design  teams  the  technical  data,  risk 
reduction  and  demonstrations  that  allowed 
them  to  assess,  and  as  appropriate,  use  this 
integrated  technology.  This  was 
accomplished  with  a  single  program  of 
integrated  technical  activities  involving 
normally  competitive  contractors  working 
together  towards  a  common  goal.  The 
information  and  demonstration  results  were 
made  available  to  all  JSF  WSCs  to 
implement  into  their  competitive  JSF 
aircraft  design,  as  applicable.  This 
approach  enabled  one  effort  to  serve  all. 

The  technical  details  and  results  of  the 
various  J/IST  maturation  activities  are 
documented  in  the  technical  reports  and 
papers  listed  in  the  suggested  reading  list. 
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